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Haynes Generating Station, Long Beach, California. Photo courtesy of Department of 


Water and Power, City of Los Angeles. 


os Angeles Department of Water 

and Power (LADWP) is the largest 

municipal utility in the United 
States. The LADWP provides water and 
electricity to more than 3.5 million resi- 
dents within the boundaries of the City of 
Los Angleses. The LADWP has approx- 
imately 1.3 million electrical customers 
and a local generating capacity of 4,300 
megawatts. Additional capacity is 
obtained via long-term energy contracts. 
Energy sources include fuel oil, natural 
gas, coal, hydro and nuclear. The Haynes 
Generating Station, located in Long 
Beach, California, is the largest of 
LADW'P's four fossil fuel plants, with six 
turbine generators producing 1600 
megawatts of electricity. LADWP-Haynes 
has had a long history of success using 
Bently Nevada products. LADWP cur- 
rently uses, or has used, Bently Nevada 
3300 and 7200 Monitoring Systems, the 
ADRE® for Windows and ADRE 3 Sys- 
tems, 108 Data Acquisition Instrument, 
Digital Vector Filter 3, Digital Vector Filter 
2, Trendmaster 2000 and Snapshot. For 
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continuous protection, all six of Haynes’ 
turbine generator sets are monitored by 
either Bently Nevada 3300 or 7200 Mor- 
itoring Systems. 


History of data collection and 
analysis at Haynes 

In the mid to late 1980s, LADWP’s 
General Services group, in downtown 
Los Angeles, ran a centralized data col- 
lection and analysis service for all 
LADWP generating stations. This pro- 
gram used portable instruments. When- 
ever the operators at Haynes suspected 
a machinery problem, the Engineering 
Section of General Services (now called 
Generation Maintenance) was asked to 
investigate. At Haynes, LADWP didn't 
have the people, expertise or equip- 
ment necessary to collect and analyze 
vibration data. Depending on the per- 
sonnel available at General Services, it 
would sometimes take days before a 
machinery analyst arrived. Unfor- 
tunately, when an analyst would arrive, 
there was little or no historical machine 
operating data available to him. 
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In 1989, LADWP management decided 
that each plant would become respons- 
ible for its own data collection and anal- 
ysis program. At first, this was more of a 
time-based than a condition-based pro- 
gram. Records were kept, but somewhat 
haphazardly. Regularly scheduled main- 
tenance was being performed, rather 
than proactively determining the mainte- 
nance that would be required at the next 
scheduled outage. Weekly vibration mea- 
surements were taken on most machines 
using a portable instrument. However, 
it was soon realized that weekly 
measurements provided too little 
data to make timely operating and 
maintenance decisions. And often, 
due to staffing constraints, these read- 
ings were only taken once per month. 

In 1990, LADWP’s Condition-Based 
Maintenance (CBM) Task Force began 
investigating online systems. Online 
monitoring promised to be more cost- 
effective than manual data collection 
and would provide more timely informa- 
tion. Timely information enhances 
LADWP's. ability to make better recom- 
mendations to management concerning 
the operation and maintenance of the 
machinery. Besides, taking readings by 
hand can be uncomfortable and unsafe 
in certain areas, for example, the inside 
of a turbine housing at 52°C (125°F). 


LADWP's Condition Based 
Maintenance (CBM) Task Force 

LADWP’s CBM Task Force began as a 
group of Haynes personnel directed by 
the plant's superintendent to develop a 
CBM handbook, LADWP recognized the 
group's value and expanded it to include 
a representative from each plant. The 
CBM Task Force now meets every six to 
eight weeks to discuss what is currently 
being done, what should be done, what is 
working and what is not working. The 
group discusses technical details and 
makes sure that everyone is on the same 
path. LADWP wants each station to have 
its own CBM program, while avoiding 
duplicate research and effort. LADWP 
management's main goals for the 
Condition-Based Maintenance program 
are: 

1. To reduce the unscheduled 

maintenance of critical 


equipment through the use of 
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eee it has also been previously 
deferred, LADWP would easily 
LADWP did not have to 


predictive and preventive 
maintenance diagnostic tech- 
niques and trending. 

2. To increase the intervals 
between maintenance for 
critical equipment, which in 
turn reduces the mainte- 
nance and capital expendi- 
tures for each generating 
station, 

One of the CBM Task Force's main 
missions is to keep informed of, and 
investigate, new technology and ways in 
which LADWP may apply it to increase 
efficiency or reduce costs. One method 
used is to invite suppliers to make a 
presentation at a Task Force meeting. 
This enables the Task Force to compare 
and contrast suppliers’ products, knowl- 
edge and support. This is how the CBM 
Task Force learned of Bently Nevada's 
Trendmaster 2000 System, 


The CBM Task Force researched 
online monitoring systems in 1990-91. 
LADWP wanted to evaluate the effective- 
ness of this type of monitoring system, so 
the CBM Task Force recommended that 
Haynes purchase and install one on a trial 





Brian White checks on Trendmaster 2000 
TIM housings, mounted near the machine. 


basis. Through a competitive evaluation 
and bid process, Bently Nevada's 
Trendmaster 2000 was chosen as the sys- 
tem that provided the most benefit at the 
lowest cost. Operators at other LADWP 
plants were to view the information pro- 
vided by this test system, either onsite at 
Haynes or remotely, and evaluate the sys- 
tem’s benefits. LADWP’s Scattergood 
Generating Station purchased a second 
Trendmaster 2000 System in 1993. 

LADWP’s experience with Bently Nev- 
ada has been good, and that's why 
LADWP uses many Bently Nevada prod- 
ucts. LADWP appreciates the way Bently 
Nevada products integrate with each 
other. Also, Bently Nevada's Design and 
Installation Services (D&IS) and Product 
Service personnel have given LADWP 
exceptional technical support during 
system installations and expansions. 
Many of LADWP’s personnel have 
attended a variety of Bently Nevada 
Technical Training Courses to increase 
their knowledge in data acquisition and 
machinery diagnostics. 


Trendmaster 2000 at Haynes 
Station 

At Haynes Station, LADWP uses the 
Trendmaster 2000 as a trending and 
diagnostic tool to detect possible prob- 
lems on the turbine generators and aux- 
iliary equipment. Because the 
Trendmaster 2000 performs periodic 
monitoring, it is likened to “taking the 
pulse” of the equipment. Bently Nevada 
3300 and 7200 Monitoring Systems pro- 
vide machinery protection on the tur- 
bine generators; the Trendmaster 2000 
provides machinery information, trend- 
ing and first level diagnostics. Alarm 
limits on the continuous monitors are 
set to the “maximum allowable” level, 
while the redundant software alarms on 
Trendmaster 2000 are set at a lower 
level. That way the Trendmaster 2000 
alarm will typically be exceeded first, 
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calculated that if just one outage were 


save twice the original investment. 
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wait long for this payback. 


providing an “early warning” signal of a 
possible machine problem, before its 
corresponding 3300 or 7200 Monitor 
relay actuates. If the Trendmaster 2000 
indicates a potential problem on a tur- 
bine generator, a portable ADRE System 
with a Data Acquisition Instrument is 
used to investigate the problem further. 
The Trendmaster 2000 System is 
LADWP's “early warning” indicator, and 
the ADRE System an advanced diagnos- 
tic tool. 

The Trendmaster 2000 also provides 
LADWP with important historical infor- 
mation on their machines. This long- 
tenm trending is something that was not 
available in the past. It is invaluable 
information for LADWP's planning of 
machinery maintenance. 

Another reason the Trendmaster 2000 
System was chosen is its Remote Access 
feature. To give operators 24-hour 
access to its data, the Trendrnaster 2000 
host computer is located in Haynes’ con- 
trol room. With a PC, modem and tele- 
phone line, LADWP engineers and 
machinery specialists can remotely 
access Haynes Station data anywhere, 
anytime. The system is typically acces- 
sed from across the plant and from 
downtown Los Angeles each morning 
and checked for any conditions that may 
have occurred overnight. 


Long-term investment 


It took LADWP approximately one 
month to complete the installation of 
the initial 200-point Trendmaster 2000 
system at Haynes. LADWP contracted 
with Bently Nevada's Design and Instal- 
lation Services group to provide site 
supervision, system checkout and sys- 
tem documentation. LADWP personne! 
installed the transducers, cabling and 
field hardware. Trendmaster 2000 sys- 
tem cable was installed in existing cable 
trays, rather than in conduit, to mini- 
mize installation cost. 
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This 5-year project began in 1992-93 
with Phase 1. In that phase, existing con- 
tinuous monitoring racks on the turbine 
generators were interfaced to the 
Trendmaster 2000. In 1993-94 under 
Phase 2, critical auxiliaries, such as boiler 
feed pumps and Induced Draft/Forced 
Draft fans, were connected to the sys- 
tem. LADWP’s goal is to put the most 
important machines online first. 


LADWP calculated that installing a 
proximity transducer, a Displacement 
Transducer Interface Module (TIM) and 
cabling on a major auxiliary has cost 
approximately $1500 per point, includ- 
ing labor and materials. On each major 
auxiliary, an XY proximity probe pair is 
installed on each bearing along with a 
single Keyphasor® probe for the 
machine. 

It has been calculated that LADWP will 
invest approximately $1.2 million to 
design, purchase and install the entire 
Haynes Generating Station vibration 
monitoring system over the 5-year pro- 
ject. However, it has also been previ- 
ously calculated that if just one outage 
were deferred, LADWP would easily save 





Bently Nevada XY 
proximity probe installation. 





twice the original investment. LADWP 
did not have to wait long for this 
payback. 


Short-term payback 


Previously, LADWP scheduled major 
machine outages based on time of ser- 
vice (the number of generating hours), 
and these averaged once every four to 
six years. At limes, upon opening a 
machine for overhaul, it was found that 
the machine did not need repairs exten- 
sive enough to warrant disassembling 
the machine. Also, sometimes machin- 
ery would return after being repaired by 
an outside vendor and would not meet 
repair specifications. This all added up 
to higher maintenance costs for LADWP. 


Deferring an outage can save a utility 
millions of dollars. LADWP is depending 
on the Trendmaster 2000 system and the 
Conditioned Based Maintenance pro- 
gram to lengthen the period between 
major overhauls. LADWP has estimated 
that in a 12 year maintenance cycle, 
approximately $2.5 million dollars per 
year will be saved through deferred out- 
ages using CBM. A typical unplanned 
outage can cost LADWP about $10 mil- 
lion dollars in lost revenue and added 
repair costs. This costs far more than a 
long-term energy contract that might be 
purchased in advance for a planned out- 
age. LADWP credits the Trendmaster 
2000 with helping to defer two outages 
since its installation in 1993., 


One example is Haynes’ Unit Four. In 
the summer of 1993, its number two 
bearing showed high vibration. It is a 
shared bearing, with an HP rotor and an 
IP, and an IP with an LP, One bearing 
supports the whole load on two turbine 
shafts, two sections of blades, and it has 
a built-in thrust bearing. It's a compli- 
cated, very large bearing. It was discov- 
ered that at different loads, the bearing 
becomes loaded or unloaded. The 
Trendmaster 2000, the ADRE 3 System 
and load testing helped LADWP to iden- 
tify this as a possible alignment problem. 
The bearing rests at a little bit of a slope, 
so at different loads, the turbine will sit 
at different axial positions. The bearing 
will shift one way and become loaded, 
then shift the other way and become 
unloaded, 
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LADWP credits the Trendmaster 2000 

with helping to defer two outages since 
its installation in 1993.” 


When unloaded, its degree of free- 
dom increases and likewise its vibration, 
up to 230 um (9 mils), which can be very 
harmful for a turbine of that size. LADWP 
tracked online data with the Trendmas- 
ter 2000 and realized that, at a load of 
about 190 megawatts, the vibration 
damped down to 100 um (4 mils). From 
190MW to 205MW, the vibration would 
increase again. From 205MW to about 
230MW, it would decrease. It was recom- 
mended that the unit be taken out of 
service and aligned. However, because 
of summer power demands, LADWP 
management decided to continue run- 
ning the unit at a load identified with low 
vibration levels using Trendmaster 2000 
data. By identifying acceptable operat- 
ing parameters, data from Trendmaster 
2000 helped LADWP extend the unit's 
run time, 

LADWP's CBM Task Force docu- 
mented this payback and reported it to 
Management in its yearly report. The 
report informed management of the pro- 
fram’s status, goals that were achieved, 
and also summarized success stories. A 
bimonthly status report is also generated 
for the Station Superintendent that 
details current machine status, problems 
identified (with supporting documenta- 
tion) and recommendations. In just its 
first months of operation, the Trendmas- 
ter 2000 has been credited by LADWP 
with helping to save money equal to the 
amount that was spent to date on the 
Haynes CBM program, through deferred 
Outages, early problem identification and 
avoided maintenance. 

LADW'P has also benefitted from using 
the Trendmaster 2000's Remote Access 
Software. A trip from the Power Operat- 
ing and Maintenance Division office in 
downtown Los Angeles to Haynes Sta- 
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tion takes about 40 minutes. With 
Trendmaster 2000, Generation Mainte- 
nance personnel no longer have to leave 
downtown L.A. to access and analyze 
data, print out plots and generate 
reports on the condition of the 
machines at Haynes. The Trendmaster 
2000's Remote Access feature results in 
minimized driving time, which in turn 
saves time and money. The Remote 
Access feature is also used to obtain 
quick verifications and recommenda- 
tions from Bently Nevada Machinery 
Diagnostics Services (MDS) Engineers 
and Product Service representatives. 
Assistance can be requested from the 
local Bently Nevada office or corporate 
headquarters, and Bently Nevada per- 
sonnel can then access Haynes’ system 
directly via modem without having to 
travel to the site. 


Success, credibility and long-term 
vision 

LADWP's CBM program is continu- 
ously gaining credibility with manage- 
ment. Haynes station management now 
relies on CBM reports and trusts the 
recommendations they make, History 
has shown that CBM team recommenda- 
tions prove themselves correct. More 
often than not, when a machine rub is 
predicted and the machine is opened 
up, clear evidence is found to support it. 
With these successes, management can 
rightfully have confidence in the team's 
recommendations. 


Since Southern California’s economy 


is currently very sluggish, LADWP man- 
agement is counting on the CBM pro- 
gram to save the utility money through 
reduced operation and maintenance 
costs, By supplying important current 
and historical condition information on 


LADWP machinery, Trendmaster 2000 
has become a big part of the CBM pro- 
gram. The information it provides helps 
LADWP to determine which machines 
need service, and the maintenance they 
require, during an outage. This enables 
LADWP to plan ahead for a faster, more 
efficient turnaround. 

LAD WP's long term plan is to interface 
all main auxiliaries to the Trendmaster 
2000. When installing the initial system, 
long-term monitoring needs were con- 
sidered and planned for. In the first 
phase, all the main cabling and cable 
T-Connectors required for future expan- 
sions were also installed. This will mini- 
mize future expansion costs, since 
cabling costs tend typically to be the 
highest installation cost factor, LADWP 
appreciates Trendmaster 2000's flex- 
ibility. With the main cabling already in, 
expansion to more machines requires 
only the installation of additional trans- 
ducers, TIMs and housings. 

LADWP management anticipates that 
the Trendmaster 2000 System will help it 
meet future maintenance challenges. A 
large challenge is determining the most 
effective time for machine maintenance: 
not before and certainly no later than 
required. To be successful in the long 
term, LADWP must become more cost- 
efficient. Trendmaster 2000 is one of the 
investments made by LADWP to help 
meet the utility’s long-term goals. 
LADWP has used “benchmarking” 
studies to compare LADWP plants with 
each other and with other utilities, to 
determine best management, mainte- 
nance and operations practices. 
LADWP’s goal is to learn which practices 
will help them succeed. LADWP is taking 
a proactive approach to developing and 
optimizing future maintenance 
strategies. W 
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